A threshold for a q-sorting methodology for computer-adaptive surveys

© 2017 Proceedings of the 25th European Conference on Information Systems, ECIS 2017. All rights reserved. Computer-Adaptive Surveys (CAS) are multi-dimensional instruments where questions asked of respondents depend on the previous questions asked. Due to the complexity of CAS, little work has been done on developing methods for validating their content and construct validity. We have created a new q-sorting technique where the hierarchies that independent raters develop are transformed into a quantitative form, and that quantitative form is tested to determine the inter-rater reliability of the individual branches in the hierarchy. The hierarchies are then successively transformed to test if they branch in the same way. The objective of this paper is to identify suitable measures and a “good enough” threshold for demonstrating the similarity of two CAS trees. To find suitable measures, we perform a set of bootstrap simulations to measure how various statistics change as a hypothetical CAS deviates from a “true” version. We find that the 3 measures of association, Goodman and Kruskal's Lambda, Cohen's Kappa, and Goodman and Kruskal's Gamma together provide information useful for assessing construct validity in CAS. In future work we are interested in both finding a “good enough” threshold(s) for assessing the overall similarity between tree hierarchies and diagnosing causes of disagreements between the tree hierarchies

A test of a computer-adaptive survey using online reviews

© 26th European Conference on Information Systems: Beyond Digitization - Facets of Socio-Technical Change, ECIS 2018. All Rights Reserved. Traditional surveys are excellent instruments for establishing the correlational relationship between two constructs. However, they are unable to identify reasons why such correlations exist. Computer-Adaptive Surveys (CAS) are multi-dimensional instruments where questions asked of respondents depend on the previous questions asked. Assessing the validity of CAS is an underexplored research area as CAS differs from traditional surveys. Therefore, validating a CAS requires different techniques. This study attempts to validate the conclusion validity of a CAS about café customer satisfaction using online customer reviews. For our CAS to have conclusion validity, there should be a high correspondence where most respondents in CAS and online reviewers both agree that certain constructs are the cause of their dissatisfaction. We created a Computer-Adaptive Survey (CAS) of café satisfaction and used online customer reviews to assess its conclusion validity. Our research thus contributes to the measurement literature in two ways, one, we demonstrate that CAS captures the same criticisms of cafes as that in online reviews, and two, CAS captures problems about customer satisfaction at a deeper level than that found in online reviews

A Q-sorting methodology for Computer-Adaptive Surveys - Style "Research"

Computer-Adaptive Surveys (CAS) are multi-dimensional instruments where questions asked of respondents depend on the previous questions asked. Due to the complexity of CAS, little work has been done on developing methods for validating their construct validity. This paper describes the process of using a variant of Q-sorting to validate a CAS item bank. The method and preliminary results are presented. In addition, lessons learned from this study are discussed

The consumer scam: an agency-theoretic approach

Despite the extensive body of literature that aims to explain the phenomenon of consumer scams, the structure of information in scam relationships remains relatively understudied. The purpose of this article is to develop an agency-theoretical approach to the study of information in perpetrator-victim interactions. Drawing a distinction between failures of observation and failures of judgement in the pre-contract phase, we introduce a typology and a set of propositions that explain the severity of adverse selection problems in three classes of scam relationships. Our analysis provides a novel, systematic explanation of the structure of information that facilitates scam victimisation, while also enabling critical scrutiny of a core assumption in agency theory regarding contract design. We highlight the role of scam perpetrators as agents who have access to private information and exercise considerable control over the terms and design of scam relationships. Focusing on the consumer scam context, we question a theoretical assumption, largely taken for granted in the agency literature, that contact design is necessarily in the purview of the uninformed principal

Computer-Adaptive Surveys (CAS) as a means of answering questions of why

© PACIS 2017. Traditional surveys are excellent instruments for establishing the correlational relationship between two constructs. However, they are unable to identify reasons why such correlations exist. Computer-Adaptive Surveys (CAS) are multi-dimensional instruments where questions asked of respondents depend on the previous questions asked. Their principal advantage is they allow the survey developer to input a large number of potential causes. Respondents then roll down through the causes to identify the one or few significant causes impacting a correlation. This study compared a café satisfaction CAS to a traditional survey of the same item bank to test whether CAS performs its intended task better than a traditional survey. Our study demonstrates that when one is trying to find root cause, CAS achieves a higher response rate, requires fewer items for respondents to answer, has better item discrimination, and has a higher agreement among respondents for each item